System and method for correcting paving mat defects

ABSTRACT

A method includes receiving sensor data indicative of a paved surface, and identifying a defect associated with the paved surface based at least in part on the sensor data. The method also includes determining that the defect is of a defect type based on determining that a value associated with the defect is within a value range associated with the defect type. The method further includes generating a command associated with the defect that, when executed by a machine, at least partially remedies the defect. The method also includes providing the command to an electronic device via a network.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application is a continuation of and claims priority to U.S.patent application Ser. No. 16/827,466, filed Mar. 23, 2020, which isfully incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to a control system for a paving machine.More specifically, the present disclosure relates to a control systemconfigured to identify paving mat defects and generate instructions tocorrect the identified paving mat defects.

BACKGROUND

A paving machine, such as an asphalt paver, is a self-propelledconstruction machine designed to receive, convey, distribute, profile,and partially compact paving material. Such a paving machine acceptsheated paving material (e.g., asphalt) into a receiving hopper at thefront of the paving machine. The heated asphalt material in the hopperis conveyed to the rear of the paving machine by conveyors positioned ata base of the hopper. The asphalt material is then distributed across awidth of the paving machine by means of two opposing screws or augers.Finally, a screed assembly located at the rear of the paving machineprofiles and compacts the asphalt material into an asphalt surface,referred to as a “mat”.

A paving machine has numerous parts that can interact with one anotherand the surroundings of the paving machine, and is subject to varioussettings based on the type of paving material being applied andcharacteristics of the environment. If a part of the paving machine ismalfunctioning, an error is present in a computing system of the pavingmachine, or a setting of the paving machine is incorrect or unsuitablefor current conditions, to name a few examples, the resulting mat mayhave one or more defects. Defects of the mat (paved surface) may requirethat the mat be repaired, or in some cases, that the mat is removed andreplaced entirely, costing valuable time and resources. Although mostoperations in the paving process rely heavily on the training, skill,and experience of paving machine operators, often times the pavingmachine operator or other personnel do not have sufficient experience toidentify every possible defect that arises in the mat, and/or how toresolve and mitigate defects once identified.

An example system for determining road surface defects is described inU.S. Patent Application No. 2012/0218441 (hereinafter referred to as the'441 reference). In particular, the system described in the '441reference is configured to analyze infrared thermal images of the roadsurface, and to diagnose defects present on the road. The '441 referencefurther describes various preventive maintenance solutions. However, thesystem described in the '441 reference requires an analyst to judge allimages that depict defects one by one to give conclusions of treatmentand/or a proposed treatment solution. The accuracy of such conclusionsdepends upon the experience and training of the individual analyst, andthese conclusions are prone to human error. Additionally, theconclusions provided by the analyst described in the '441 reference aredependent upon the analyst being available at the time a paving defectis identified, which in a paving scenario, may cause delay based onanalyst bandwidth and/or scheduling. Such delays reduce the overallefficiency of the system and/or the paving operation.

Example embodiments of the present disclosure are directed towardovercoming the deficiencies described above.

SUMMARY

In an aspect of the present disclosure, an example method includesreceiving, with a controller, image data indicative of a paved surfaceand identifying, with the controller and based at least in part on theimage data, a defect associated with the paved surface, the defecthaving a characteristic that is characterized by a value. Such anexample method also includes determining, with the controller, that thevalue is within a value range corresponding to a defect type, anddetermining, based at least in part on the value being within the valuerange, that the defect is of the defect type. Such an example methodfurther includes generating a command associated with the defect andbased at least in part on determining that the defect is of the defecttype. In such methods, the command identifies a proposed machineactivity which, when executed by a machine on the paved surface, atleast partially remedies the defect. Such an example method furtherincludes providing the command to an electronic device via a network.

In another aspect of the present disclosure, a system includes a sensingdevice and a system controller in communication with a controller of thesensing device. The system controller is configured to receive, from thesensing device, sensor data indicative of a paved surface, and identify,based at least in part on the sensor data, a defect associated with thepaved surface, the defect having a characteristic that is characterizedby a value. The controller is also configured to determine that thevalue is within a value range corresponding to a defect type, anddetermine, based at least in part on the value being within the valuerange, that the defect is of the defect type. The controller is furtherconfigured to generate a command associated with the defect and based atleast in part on determining that the defect is of the defect type,wherein the command identifies a proposed machine activity which, whenexecuted by a machine on the paved surface, at least partially remediesthe defect. The controller is further configured to provide the commandto an electronic device and via a network.

In yet another aspect of the present disclosure, an example methodincludes receiving, with a controller and from a sensing device locatedat a worksite via a network, sensor data indicative of a paved surfaceat the worksite, and identifying, based at least in part on the sensordata, a defect associated with the paved surface, where the defect has acharacteristic that is characterized by a value. Such an example methodfurther includes determining, with the controller, that the value iswithin a value range corresponding to a defect type. Such an examplemethod further includes determining, with the controller, and based atleast in part on the value being within the value range, that the defectis of the defect type. Such an example method further includesgenerating, with the controller, a command associated with the defectand based at least in part on determining that the defect is of thedefect type, where the command identifies a proposed machine activitywhich, when executed by a paving machine on the paved surface, at leastpartially remedies the defect. Such an example method further includesproviding the command with the controller to the paving machine via thenetwork, where the command causes the paving machine to change a settingof a component of the paving machine associated with the proposedmachine activity.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a perspective view of a paving machine that includes a machinecontroller in communication with a mat defect identification componentin accordance with an example of the present disclosure.

FIG. 2 is a block diagram schematically representing a control systemassociated with the paving machine in accordance with an example of thepresent disclosure.

FIG. 3 is a schematic illustration of a paving machine depositing pavingmaterial on a paving surface, where the mat is free from defects, inaccordance with an example of the present disclosure.

FIG. 4 is a schematic illustration of an image of a mat on a pavingsurface with a defect, and commands that may be provided by a controlsystem based on the defect, in accordance with an example of the presentdisclosure.

FIG. 5 is a schematic illustration of a paving machine and associatedcomponents of the paving machine depositing paving material on a pavingsurface, and dimensions that may be altered to adjust how the pavingmaterial is deposited on the paving surface, in accordance with anexample of the present disclosure.

FIG. 6 is a schematic illustration of a sensor included on a pavingmachine that determines a distance to paving material, in accordancewith an example of the present disclosure.

FIG. 7 is another schematic illustration of an image of a mat on apaving surface with a defect, and commands that may be provided by acontrol system based on the defect, in accordance with an example of thepresent disclosure.

FIG. 8 is yet another schematic illustration of an image of a mat on apaving surface with a defect, and commands that may be provided by acontrol system based on the defect, in accordance with an example of thepresent disclosure.

FIG. 9 is still another schematic illustration of an image of a mat on apaving surface with a defect, and commands that may be provided by acontrol system based on the defect, in accordance with an example of thepresent disclosure.

FIG. 10 is an additional schematic illustration of an image of a mat ona paving surface with a defect, and commands that may be provided by acontrol system based on the defect, in accordance with an example of thepresent disclosure.

FIG. 11 is a further schematic illustration of an image of a mat on apaving surface with a defect, and commands that may be provided by acontrol system based on the defect, in accordance with an example of thepresent disclosure.

FIG. 12 is one more schematic illustration of an image of a mat on apaving surface with a defect, and commands that may be provided by acontrol system based on the defect, in accordance with an example of thepresent disclosure.

FIG. 13 is a flow chart depicting a method of generating a command toremedy an identified paving mat defect in accordance with an example ofthe present disclosure.

DETAILED DESCRIPTION

Wherever possible, the same reference numbers will be used throughoutthe drawings to refer to the same or like parts. FIG. 1 shows a pavingmachine 100 which is used, for example, to deposit asphalt, concrete, orother materials on a work surface associated with a worksite. As shownin FIG. 1, the paving machine 100 is in communication with one or morecomputing devices 204 via a network 206. The one or more computingdevices 204 include a mat defect ID component 214 configured to identifydefects in a paving mat and generate commands that, when executed by thepaving machine 100, a crew member at a worksite with the paving machine100, a haul truck delivering paving material 110 to the paving machine100, and so forth, at least partially remedy the defect. While thefollowing detailed description describes examples in connection with thepaving machine 100, it should be appreciated that the descriptionapplies equally to the use of the present disclosure in other machinesas well.

The paving machine 100 includes a tractor portion 102 supported on a setof ground-engaging elements 104. The tractor portion 102 includes atractor frame 106, as well as a power source for driving theground-engaging elements 104. Although the ground engaging elements 104are illustrated as continuous tracks, it should be contemplated that theground engaging elements 104 may be any other type of ground engagingelements as well, for example, wheels etc. In some cases the powersource is a conventional internal combustion engine operating on fossilor hybrid fuels, or in examples is an electrically operated drivepowered by alternate energy sources. The paving machine 100 includes ahopper 108 for storing a paving material 110. The paving machine 100also includes a conveyor system 112 for conveying the paving material110 from the hopper 108 to other downstream components of the pavingmachine 100. For example, the paving machine 100 includes an augerassembly 114 which receives the paving material 110 supplied via theconveyor system 112, and distributes the paving material 110 on thepaving surface 122. In some examples, the auger assembly 114 includes atleast one main auger. In some cases, the auger assembly 114 includes amain auger, and an auger extension coupled to the main auger via anauger bearing or other coupling component. Additionally, in someexamples, the auger assembly 114 includes a main auger and an additionalauger disposed opposite the main auger. In such examples, the main augerand the additional auger is configured to distribute the paving material110 across substantially an entire width of the paving machine 100. Thepaving machine 100 further includes a tow arm 116 which couples a heightadjustable screed portion 118 to the tractor portion 102 so as to spreadand compact the paving material 110 into a mat 120 on the paving surface122. The tow arm 116 is actuated by a hydraulic actuator, an electricactuator (not shown), and/or any other type of actuator as perapplication requirements. In examples, any of the ground-engagingelements 104, the hopper 108, the conveyor system 112, the augerassembly 114, the tow arm 116, and/or the screed portion 118 can receivecommands from the mat defect ID component 214 to adjust settings such asspeed, height, and the like to remedy defects detected in the mat 120.

Further referring to FIG. 1, an operator station 124 is coupled to thetractor portion 102. The operator station 124 includes a console 126 andother levers or controls (not shown) for operating the paving machine100. For example, the console 126 includes a control interface 128 forcontrolling various functions of the paving machine 100. The controlinterface 128 may comprise an analog, digital, and/or touchscreendisplay, and such a control interface 128 is configured to display, forexample, commands that when executed by the paving machine 100, remedydefects of mat 120 according to the present disclosure. The controlinterface 128 also supports other allied functions, including forexample, sharing various operating data with one or more other machines(not shown) operating in consonance with the paving machine 100.

As shown, the operator station 124 includes a roof 130. A communicationdevice 132 and/or a camera 134 (or other image capture device) arecoupled to the roof 130 as illustrated in FIG. 1. Alternatively, in someexamples, at least one of the communication device 132 and/or the camera134 are coupled to other portions of the paving machine 100. Thecommunication device 132 is capable of determining a location of thepaving machine 100, and includes and/or comprises a component of aglobal positioning system (GPS). For example, the communication device132 comprises a GPS receiver, transmitter, transceiver, and/or othersuch device, and the communication device 132 is in communication withone or more GPS satellites (not shown) to determine a location of thepaving machine 100 continuously, substantially continuously, or atvarious time intervals. Alternatively or additionally, the communicationdevice 132 is in communication with a ground-based location station,virtual reference station (VRS), global navigation satellite system, orother survey coordinate system to determine the location of the pavingmachine 100. In examples, the communication device 132 also enables thepaving machine 100 to communicate with the one or more other machines,and/or with one or more remote servers, processors, or control systemslocated remote from the worksite at which the paving machine 100 isbeing used, such as the computing devices 204 that include the matdefect ID component 214. In some examples, the camera 134 is a digitalcamera configured to record and/or transmit digital images and/or videoof the mat 120, paving surface 122, and/or worksite in real-time. Insome cases, the camera 134 comprises an infrared sensor, a thermalcamera, or other like device configured to record and/or transmitthermal images of the mat 120, paving surface 122, and/or worksite inreal-time. In examples, the camera 134 provides image data to the matdefect ID component 214 via the network 206 (e.g., by way of thecontroller 154), which the mat defect ID component 214 uses to identifydefects of the mat 120 based on the image data.

As shown in FIG. 1, the paving machine 100 also includes one or moretemperature sensors 136, 138, 140. One or more such temperature sensors136, 138, 140 may comprise a mechanical, electrical, electro-mechanical,electronic, infrared, or any other type of a temperature sensor known inthe art. In some examples, one or more such temperature sensors 136,138, 140 include an air purge device or other such device (not shown) toprevent debris from forming on the temperature sensor. Such an exampleair purge device receives purge air from an air source storingpressurized air to prevent any dirt, debris etc. which may stick to thetemperature sensor.

In some examples, the temperature sensor 136 comprises a firsttemperature sensor connected to the paving machine 100 proximate theauger assembly 114, and the temperature sensor 136 is configured tosense, measure, detect, and/or otherwise determine temperatures of thepaving material 110 at the auger assembly 114. As shown in FIG. 1, thepaving machine 100 includes a first side 142 (e.g., a right-hand side)and a second side 144 (e.g., a left-hand side) opposite the first side142. Likewise, in such examples the auger assembly 114 includes a firstportion 146 disposed on the first side 142 of the paving machine 100,and a second portion 148 (labeled in FIG. 1 but obstructed from view)disposed on the second side 144 of the paving machine 100. In suchexamples, the first portion 146 of the auger assembly 114 includes afirst main auger extending laterally from proximate a center of theauger assembly 114. The first portion 146 of the auger assembly 114 alsoincludes a first auger extension connected to the first main auger via afirst auger bearing or other coupling component. Further, in suchexamples the second portion 148 of the auger assembly 114 includes asecond main auger, opposite the first main auger, extending laterallyfrom proximate the center of the auger assembly 114. The second portion148 of the auger assembly 114 also includes a second auger extensionconnected to the second main auger via a second auger bearing or othercoupling component. In such examples, the first main auger issubstantially coaxially aligned with the second main auger to assist inevenly depositing the paving material 110 on the paving surface 122and/or across substantially an entire width of the paving machine 100.

In examples, the temperature sensor 136 is positioned proximate thefirst portion 146 of the auger assembly 114, and is configured todetermine the temperature of the paving material 110 at the firstportion 146 of the auger assembly 114. Further, at least one additionaltemperature sensor (not shown) is positioned proximate the secondportion 148 of the auger assembly 114 on the second side 144 of thepaving machine 100. In such examples, the at least one additionaltemperature sensor is configured to determine the temperature of thepaving material 110 at the second portion 148 of the auger assembly 114.In such examples, the temperature sensor 136 is positioned and/orotherwise configured to determine the temperature of the paving material110 at and/or proximate the first auger bearing described above, or atother locations associated with the first portion 146 of the augerassembly 114. Similarly, in such examples the at least one additionaltemperature sensor described above is positioned and/or otherwiseconfigured to determine the temperature of the paving material 110 atand/or proximate the second auger bearing described above, or at otherlocations associated with the second portion 148 of the auger assembly114. For example, the temperature sensor 136 and/or the additionaltemperature sensor comprises infrared sensors configured to sense,measure, and/or detect infrared radiation emitted by the paving material110 as the paving material 110 passes the first and second augerbearings, and/or just before the paving material 110 is processed by thescreed portion 118. The temperature sensor 136 generates a signalincluding information indicative of the temperature of the pavingmaterial 110 at the first portion 146 of the auger assembly 114.Likewise, the additional temperature sensor described above generates asignal including information indicative of the temperature of the pavingmaterial 110 at the second portion 148 of the auger assembly 114. Insome examples, the temperature sensor 136 provides the signal (e.g., byway of the controller 154) including information indicative of thetemperature of the paving material 110 to the mat defect ID component214, which uses information included in the signal to identify defectsof the mat 120.

With continued reference to FIG. 1, the temperature sensor 138 ispositioned on the tractor frame 106, the hopper 108, and/or at any otherlocation on the paving machine 100 convenient for determining atemperature of the paving surface 122. For example, the temperaturesensor 138 is positioned on the tractor frame 106 proximate the groundengaging elements 104, and/or at any other location convenient fordetermining a ground temperature. The temperature sensor 138 isconfigured to generate a signal including information indicative of theground temperature. Additionally, the temperature sensor 140 ispositioned on the roof 130, the tractor portion 102, and/or at any otherlocation on the paving machine 100 convenient for determining an ambienttemperature at the worksite. The temperature sensor 140 is alsoconfigured to determine an ambient pressure and/or other parameters atthe worksite. In such examples, the temperature sensor 140 generates asignal including information indicative of ambient conditions such asambient temperature, pressure, etc. The temperature sensor 138 and/orthe temperature sensor 140 provide the respective signals (e.g., by wayof the controller 154) including information indicative of thetemperatures of the paving surface 122 or the ambient temperature of theworksite to the mat defect ID component 214, which uses informationincluded in the signal(s) to identify defects of the mat 120 and/or togenerate commands as described in more detail below.

In some examples, the paving machine 100 includes a LIDAR sensor 150and/or a RADAR sensor 152. While depicted on the roof 130, the LIDARsensor 150 and/or the RADAR sensor 152 may be positioned at any locationon the paving machine 100, or another location proximate the pavingmachine 100 at which the sensors capture sensor data of the mat 120.Additionally, while one LIDAR sensor 150 and one RADAR sensor 152 areshown, any number of LIDAR sensors and/or RADAR sensors may beincorporated into and/or proximate the paving machine 100. In examples,the LIDAR sensor 150 measures distance to a target (e.g., the mat 120)by illuminating the target with laser light and measuring the reflectedlight with a sensor. The LIDAR sensor 150 uses differences in laserreturn times and wavelengths to make digital 3D representations of thetarget. For instance, the LIDAR sensor 150 generates a signal includinginformation indicative of a 3D representation of the target.Additionally, in some examples, the RADAR sensor 152 uses radio waves todetermine the range, angle, or velocity of a target (e.g., the mat 120).The RADAR sensor 152 generates a signal that includes information suchas range, angle, and/or velocity of the target. It should becontemplated that the paving machine 100 may include various othersensors to measure various other parameters related to the pavingmachine 100, the mat 120, and/or the worksite. The LIDAR sensor 150and/or the RADAR sensor 152 provide the respective signals (e.g., by wayof the controller 154) including information indicative of theenvironment of the worksite to the mat defect ID component 214, whichuses information included in the signal(s) to identify defects of themat 120 and/or to generate commands as described in more detail below.

The paving machine 100 also includes a controller 154 in communicationwith the control interface 128, the communication device 132, the camera134, the temperature sensors 136, 138, 140, the LIDAR sensor 150, theRADAR sensor 152 and/or other components of the paving machine 100. Thecontroller 154 may be a single controller or multiple controllersworking together to perform a variety of tasks. In examples, thecontroller 154 embodies a single or multiple microprocessors, fieldprogrammable gate arrays (FPGAs), digital signal processors (DSPs),and/or other components configured to identify, calculate, and/orotherwise determine defects of the mat 120 based on one or more signalsreceived from the camera 134, the temperature sensors 136, 138, 140, theLIDAR sensor 150, and/or the RADAR sensor 152. Numerous commerciallyavailable microprocessors can be configured to perform the functions ofthe controller 154. Various known circuits may be associated with thecontroller 154, including power supply circuitry, signal-conditioningcircuitry, actuator driver circuitry (i.e., circuitry poweringsolenoids, motors, or piezo actuators), and communication circuitry. Insome examples, the controller 154 is positioned on the paving machine100, while in other examples the controller 154 is positioned at anoff-board location and/or remote location relative to the paving machine100. The present disclosure, in any manner, is not restricted to thetype of controller 154 or the positioning of the controller 154 relativeto the paving machine 100.

FIG. 2 is a block diagram schematically representing a control system200 associated with the paving machine 100 in accordance with an exampleof the present disclosure. In any of the examples described herein, thecontrol system 200 includes at least one of the controller 154, thecontrol interface 128, the communication device 132, the camera 134, thetemperature sensor 136, the additional temperature sensor (not shown)described above with respect to the second portion 148 of the augerassembly 114, the temperature sensor 138, the temperature sensor 140,the LIDAR sensor 150, the RADAR sensor 152, and/or any other sensors orcomponents of the paving machine 100. In such examples, the controller154 is configured to receive respective signals from such components.For example, the controller 154 receives one or more signals from thecommunication device 132 including information indicating a location ofthe paving machine 100. As shown in FIG. 2, the communication device 132is connected to and/or otherwise in communication with one or moresatellites 202 or other GPS components configured to assist thecommunication device 132 in determining the location of the pavingmachine 100. In some examples, such satellites 202 or other GPScomponents comprise components of the control system 200. The controller154, the control interface 128, the communication device 132, the camera134, the temperature sensor 136, the additional temperature sensor (notshown) described above with respect to the second portion 148 of theauger assembly 114, the temperature sensor 138, the temperature sensor140, the LIDAR sensor 150, the RADAR sensor 152, and/or any othersensors or components of the paving machine 100 provide informationusable to determine defects in the mat 120 and generate commands toprovide to the paving machine 100 to at least partially remedy thedefects.

As shown in FIG. 2, the control system 200 may also include one or moreadditional components. For example, the control system 200 includes oneor more remote servers, processors, or other such computing devices 204.Such computing devices 204 may comprise, for example, one or moreservers, laptop computers, or other computers located at a pavingmaterial plant remote from the worksite at which the paving machine 100is being used. In such examples, the communication device 132 isconnected to and/or otherwise in communication with such computingdevices 204 via a network 206. The network 206 may be a local areanetwork (“LAN”), a larger network such as a wide area network (“WAN”),or a collection of networks, such as the Internet. Protocols for networkcommunication, such as TCP/IP, may be used to implement the network 206.Although embodiments are described herein as using a network such as theInternet, other distribution techniques may be implemented that transmitinformation via memory cards, flash memory, or other portable memorydevices. The control system 200 may further include one or moreelectronic devices such as tablets, mobile phones, laptop computers,and/or other mobile devices 208. Such mobile devices 208 may be locatedat the worksite or, alternatively, one or more such mobile devices 208may be located at the paving material plant described above, or atanother location remote from the worksite. In such examples, thecommunication device 132 is connected to and/or otherwise incommunication with such mobile devices 208 via the network 206. In anyof the examples described herein, the sensor data, location information,paving material maps, and/or any other information received, processed,or generated by the controller 154 is provided to the computing devices204 and/or the mobile devices 208 via the network 206.

In some examples, the controller 154 receives signals from one or moreof the camera 134, the temperature sensor 136, the additionaltemperature sensor described above with respect to the second portion148 of the auger assembly 114, the temperature sensor 138, thetemperature sensor 140, the LIDAR sensor 150, and/or the RADAR sensor152. For instance, the controller 154 receives sensor data such asimages, video, LIDAR data, RADAR data, infrared data, and/or othersensor data associated with the mat 120. Alternatively or additionally,the mobile device 208 includes a camera 210 (and/or any of the othersensors described herein) that generates sensor data, such as sensordata associated with the mat 120. For instance, a crew member operatingthe paving machine 100 may suspect that the mat 120 includes a defect,and capture an image of the suspected defect in the mat 120 using thecamera 210 of the mobile device 208. In some examples, the crew memberutilizes an interactive mat defect component 212 of the mobile device208 to communicate the image to the computing device 204 via the network206. The interactive mat defect component 212 is, in some cases, anapplication installed on the mobile device 208 that enables a user, suchas the crew member just described, to submit sensor data associated withsuspected mat defects, compare the sensor data to sensor data known tobe associated with a particular defect, receive commands on how tocontrol the paving machine 100 to remedy the defect, and so forth asdescribed herein.

The controller 154 and/or the interactive mat defect component 212 ofthe mobile device 208 provides sensor data to a mat defectidentification (ID) component 214. The mat defect ID component 214 isconfigured to identify mat defects from sensor data, and map the matdefect to one or more commands that will at least partially remedy themat defect. For example, the mat defect ID component 214 includes adefect database 216 that includes images or other sensor data types thatare known to illustrate mat defects and the defect types of the matdefects. The defect database 216, in some cases, includes sensor dataillustrating different defects in different weather conditions, atdifferent times of day, at different ambient temperatures, havingdifferent paving material properties, and so forth. Additionally, inexamples, the defect database 216 includes maps to commands associatedwith defects that at least partially remedy the respective defects. Insome cases, multiple different commands are mapped to a single defect,where the multiple different commands may be applied together to remedythe defect (e.g., increase the grade reference and tighten the screedplates), and/or may be applied sequentially to remedy the defect (e.g.,increase the grade reference, and if the defect is not remedied, tightenthe screed plates).

In the control system 200, the mat defect ID component 214 includes afeature detector 218 configured to compute abstractions of image (orother sensor) information and make local decisions at image points inthe image regarding whether the image point includes an image feature ornot. For instance, the feature detector 218 is configured to detectedges, corners, ridges, blobs, or other image feature types. An edge canbe characterized by a gradient magnitude of image brightness that isgreater than a threshold between pixels of an image, where the pixelsmay or may not be proximate one another. A corner is characterizedsimilarly to an edge, but is further analyzed to determine rapid changesin direction of the edge, and thus has a two-dimensional structure. Aridge corresponds to an elongated feature in an image represented bycapturing the major axis of symmetry of the feature, while an edgetypically relates to a boundary of an object. Blobs provide informationon a region of interest in an image by implementing image structures,which in some cases may provide more accuracy than corners, that aremore point-like. For instance, a detected blob may include areas in animage that are too smooth to be detected by a corner detector. In someexamples, the mat defect ID component 214 selects a region of interestof an image (e.g., corresponding to the mat 120) on which the featuredetector 218 is to perform feature detection. However, in some cases,the feature detector 218 detects features in an image prior to thelocation of the mat defect ID component 214 determining the region ofinterest. Other feature detectors are also considered.

In some examples, the mat defect ID component 214 uses the featuresidentified by the feature detector 218 to locate, within an image, alocation of a potential mat defect of the mat 120. In an illustrativeexample, the mat defect ID component 214 detects an edge in an imagewhere a difference in extender height from a height of the screedportion 118 of the paving machine 100 has caused the mat 120 to bedifferent thicknesses. A relatively straight line may be represented inthe image that defines a boundary between the first mat thicknesscreated by the screed portion 118 and the second mat thickness createdby the extender of the paving machine 100. The feature detector 218detects such a line as, for example, an edge, and the mat defect IDcomponent 214 uses the location of the detected edge in the image tocompare the suspected defect to known defects stored in the defectdatabase 216.

For instance, the mat defect ID component 214 includes a comparisoncomponent 220 that is configured to compare a potential mat defect insensor data (such as an image) to known defects in the defect database216. In examples, the comparison component 220 determines a valueassociated with the detected feature in the sensor data, and comparesthe value to known values (and/or value ranges) associated the knowndefects in the defect database 216. In cases where image data is used,the comparison component 220 determines whether a size of a first defectin image data received from the camera 134 and/or the camera 210 iswithin a threshold size of a second defect known to be of the defecttype indicated in second image data stored in the defect database 216.The comparison component 220 relies on one or more of a variety ofcharacteristics of defects and their associated values to determine thepresence of a mat defect from sensor data. For example, characteristicsof the defects described herein include differences of materialstructure from an ideal mat 120, such as bumps, cracks, crevices,segregation of aggregate material, and so forth. Additionally, valuescharacterizing such example characteristics include size values (e.g.,length, width, height, depth, and the like of the suspected defect),temperature values, distance values and/or angle values (e.g., relativeto the paving machine 100 or a particular part of the paving machine100, relative to an extent of the mat 120, and the like), and so on. Insome examples, the defect database 216 includes value ranges associatedwith characteristics that indicate the presence (and/or absence) of adefect in the mat 120. In an illustrative example, the defect database216 includes a threshold size of large aggregate clusters that, whenpresent, indicate unacceptable segregation particles of the aggregate inthe mat 120. For example, if a threshold size in the defect database 216is equal to 2 square feet, and if a size of a cluster of large aggregateparticles is greater than 2 square feet on the mat 120, the comparisoncomponent 220 determines that a defect is present in the mat 120. Basedat least in part on this determination, the comparison component 220also determines that the defect corresponds to a defect type ofunacceptable segregation of the large and small aggregate particles.

The mat defect ID component 214 also includes a command component 222configured to generate commands associated with the defect and based ona determination that an identified defect is of a particular defecttype. In examples, the command component 222 generates the command toidentify a proposed machine activity which, when executed by the pavingmachine 100 (or other machine) on the paved surface, at least partiallyremedies the defect. For instance, the command component 222 maps thedefect type to one or more commands stored in the defects database 216known to at least partially remedy defects of the defect type. Thecommand component 222 provides the command to mobile device 208 and/orthe paving machine 100 via the network 206. In some cases, the commandcomponent 222 provides the command to the paving machine 100 such thatthe command causes the paving machine 100 to execute the command toremedy the defect without user intervention. For example, the commandcomponent 222 may receive an indication that the defect in the mat 120is segregation of the aggregate, and accordingly generate a command thatcauses the paving machine 100 to adjust a height of the auger assembly114 automatically and without user intervention to remedy the aggregatesegregation. The paving machine 100, in such examples, is asemi-autonomous or fully autonomous machine configured to navigate theenvironment and/or execute paving tasks without input from a humanoperator.

In some examples, the mat defect ID component 214 further includes oneor more machine-learned models 224 that are configured to perform tasksrelated to defect identification in images, defect type identification,command determination, and so forth. For instance, the feature detector218 utilizes a machine-learned model 224 (e.g., scale-invariant featuretransform (SIFT), histogram of oriented gradients (HOG), etc.) followedby a support vector machine (SVM) to classify objects depicted in imagesreceived from the camera 210 and/or the camera 134. Alternatively oradditionally, the feature detector 218 may utilize a machine-learnedmodel 224 that utilizes a deep learning approach based on aconvolutional neural network (CNN) to classify objects depicted inimages received from the camera 210 and/or the camera 134. For example,the feature detector 218 inputs at least a portion of an image receivedfrom the camera 210 and/or the camera 134 into a machine-learned model224 trained to identify paving defects in images, such as one of themachine-learned models just described. The feature detector 218 receivesa location of the defect in the image, and provides the location of thedefect in the image to the comparison component 220, which uses thelocation of the defect in the image to determine a defect type of thedefect.

Alternatively or additionally, the command component 222 utilizes amachine-learned model 224 to determine a command to provide to anelectronic device to remedy the defect. For example, the commandcomponent 222 may utilize a machine-learned model 224 such as aregression algorithm (e.g., ordinary least squares regression (OLSR),linear regression, logistic regression, stepwise regression,multivariate adaptive regression splines (MARS), locally estimatedscatterplot smoothing (LOESS)), a decision tree algorithm (e.g.,classification and regression tree (CART), iterative dichotomiser 3(ID3), Chi-squared automatic interaction detection (CHAID), decisionstump, conditional decision trees), or other machine-learning algorithmto determine which remedies apply to which defects, an order of how toapply different remedies to resolve a defect, which remedies to apply incombination to resolve a defect, and so forth.

The machine-learned models 224 available to the mat defect ID componentshould not be limited to those just described. For instance, themachine-learned models 224 may be part of a neural network, which is abiologically inspired algorithm which passes input data through a seriesof connected layers to produce an output. Each layer in a neural networkcan also comprise another neural network, or can comprise any number oflayers (whether convolutional or not). As can be understood in thecontext of this disclosure, a neural network can utilize machinelearning, which can refer to a broad class of such algorithms in whichan output is generated based on learned parameters.

Although discussed in the context of neural networks, any type ofmachine learning can be used consistent with this disclosure. Beyondthose just described, machine learning algorithms can include, but arenot limited to, instance-based algorithms (e.g., ridge regression, leastabsolute shrinkage and selection operator (LASSO), elastic net,least-angle regression (LARS)), Bayesian algorithms (e.g., naïve Bayes,Gaussian naïve Bayes, multinomial naïve Bayes, average one-dependenceestimators (AODE), Bayesian belief network (BNN), Bayesian networks),clustering algorithms (e.g., k-means, k-medians, expectationmaximization (EM), hierarchical clustering), association rule learningalgorithms (e.g., perceptron, back-propagation, hopfield network, RadialBasis Function Network (RBFN)), deep learning algorithms (e.g., DeepBoltzmann Machine (DBM), Deep Belief Networks (DBN), ConvolutionalNeural Network (CNN), Stacked Auto-Encoders), Dimensionality ReductionAlgorithms (e.g., Principal Component Analysis (PCA), PrincipalComponent Regression (PCR), Partial Least Squares Regression (PLSR),Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit,Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA),Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis(FDA)), Ensemble Algorithms (e.g., Boosting, Bootstrapped Aggregation(Bagging), AdaBoost, Stacked Generalization (blending), GradientBoosting Machines (GBM), Gradient Boosted Regression Trees (GBRT),Random Forest), SVM (support vector machine), supervised learning,unsupervised learning, semi-supervised learning, etc. Additionalexamples of architectures include neural networks such as ResNet50,ResNet101, VGG, DenseNet, PointNet, and the like.

While generally described above in relation to a single instance ofsensor data received from the sensors of the paving machine 100 or themobile device 208, examples are considered in which the mat defect IDcomponent 214 utilizes multiple instances of sensor data to identify amat defect and/or to generate a command to remedy a mat defect. Forexample, the mat defect ID component 214 may receive sensor data fromdifferent types of sensors. In an illustrative example, the mat defectID component 214 may receive image data from the camera 134 that isindicative of the mat 120, as well as receive temperature data from oneor more of the temperature sensors 136, 138, 140 that is indicative ofthe mat 120. The mat defect ID component 214 leverages both of thesensor data types to identify the defect of the mat 120, such as by thecomparison component 220 comparing values of the sensor data types toknown values of corresponding sensor data types associated withdifferent mat defects stored in the defect database 216. Additionally,in some examples, the command component 222 generates a command toprovide to the electronic device to remedy the defect based on thedifferent sensor data types. For instance, the command component 222 maygenerate a command that instructs the paving machine 100 to change aspeed of the auger rather than changing a temperature of the screed toremedy the defect, based on a determination that the temperature sensordata associated with the screed is within an acceptable range.

Alternatively or additionally, the mat defect ID component 214 may relyon multiple instances of a same type of sensor data to identify a defectand/or to generate a command to remedy the defect. For instance, the matdefect ID component 214 may receive multiple instances of image data,such as a first image from the mobile device 208, and a second imagefrom the camera 134 associated with the paving machine 100, and/ormultiple images from different cameras mounted on the paving machine100, to name a few examples. In some cases, first image data isgenerated by a first image capture device and second image data isgenerated by a second image capture device substantially simultaneouslywith the first image data. In other words, multiple instances of sensordata, such as images, are captured substantially simultaneously, buthave a different perspective of the mat 120. The mat defect ID component214 uses additional perspectives of the mat 120 to identify mat defectsthat may not be present in one single image of the mat, and/or to refinelocations of features identified by the feature detector 218, forinstance. As such, the feature detector 218 identifies features inadditional images and locations of the features within the additionalimages. The comparison component 220 compares suspected defects at thelocations of the identified features in the multiple images, and canincrease confidence in a determined defect type of the defect based onhaving multiple images of the defect from different perspectives.

In some examples, the mat defect ID component 214 solicits user input indetermining a defect type of a suspected mat defect, such as via theinteractive mat defect component 212. For instance, the mat defect IDcomponent 214 determines that a value associated with the defect iswithin a value range of two (or more) different defect types. The matdefect ID component 214 generates a notification comprising a firstimage from the defect database 216 corresponding to the first defecttype, a second image from the defect database 216 corresponding to thesecond defect type, and so on based on the number of defect types thatthe value falls into the respective value ranges. Additionally, thenotification includes a request for a user, such as a crew member at theworksite of the paving machine 100, to select which image is moresimilar to the defect currently being encountered at the paving machine100. The mat defect ID component 214 provides the notification via thenetwork 206 to the interactive mat defect component 212 of the mobiledevice 208, which in turn displays the notification including the imagesand the request to the user. The interactive mat defect component 212enables the user to select which image is more similar to the defectbeing encountered at the paving machine 100, such as by a touch input onone of the images in a user interface of the mobile device 208. Theinteractive mat defect component 212 provides the selection receivedfrom the user to the mat defect ID component 214 via the network 206.Upon receiving the selection of the image associated with one of thedefect types, the command component 222 generates the command based onthe selection of the defect type. The mat defect ID component 214 storesthe selection of the particular defect type with the original image orsensor data in the defect database 216, which may be used for subsequentmat defect identifications, to train the machine-learned models 224, togenerate subsequent notifications as just described, and the like.

Furthermore, in some cases, the mat defect ID component 214 storessensor data received from the sensors of the paving machine 100 and/orthe mobile device 208 in association with identified mat defects andknown defect types in the defects database 216. In this way, the defectsdatabase 216 grows to accommodate sensor data received in differentconditions, at different times of day, using different paving materials,and so forth, and thus the mat defect ID component 214 becomes moreaccurate at identifying mat defects and generating appropriate commandsto remedy the mat defects. For example, the mat defect ID component 214determines that a first defect indicated in first image data is of aparticular defect type. The mat defect ID component 214 then uses thatfirst image in subsequent comparisons of images of suspected mat defectslater in time. For instance, the mat defect ID component 214 determinesthat a second image is indicative of a defect of the particular defecttype based on a similarity of the second defect as indicated by thefirst image to the first defect as indicated by the first image.

FIG. 3 is a schematic illustration 300 of the paving machine 100depositing paving material 110 on the paving surface 122. In theschematic illustration 300, the mat 120 is free from defects, asrepresented by a generally uniform appearance of the mat 120. Inparticular, the mat 120 comprises a uniform texture from a left side 302of the mat 120 to a right side 304 of the mat 120, with no blemishes ortexture differences. In some examples, the mat defect ID component 214stores sensor data corresponding to a mat 120 that is free from defectsin the defect database 216, such as image data, temperature data,infrared data, and the like. The comparison component 220 comparessensor data corresponding to the mat 120 that is known to be free fromdefects stored in the defect database 216 to sensor data received fromsensors of the paving machine 100 and/or the mobile device 208 todetermine if subsequently-laid mats are free from defects as well. Forexample, the mat defect ID component 214 compares a size of a crack inthe mat 120 represented in image data to a threshold size (e.g., 2inches, 6 inches, 12 inches, etc.). If the crack is smaller than thethreshold size, the mat defect ID component 214 determines that at leastthat portion of the mat 120 is free from defects. If the mat defect IDcomponent 214 concludes that the mat 120 is free from defects, the matdefect ID component 214 prevents a command from being generated, and/orprevents a command from being provided to an electronic device asotherwise described herein.

FIG. 4 is a schematic illustration of an image 400 indicating a mat 120with a defect 402. In this example, the defect 402 corresponds tosegregation of large and small aggregates (e.g., rock) in the pavingmaterial 110. As noted above, segregation may result from a variety ofcauses, such as a position of a feed sensor, auger height, auger speed(e.g., revolutions per minute (RPM)), conveyor speed, augers reversing,and/or location or orientation of deflector plates, among others.

In the example illustrated in FIG. 4, the mat defect ID component 214receives the image 400 (and/or other sensor data corresponding to themat 120, as described above), and uses the feature detector 218 todetect, for instance, blobs corresponding to locations of the defect 402in the image 400. The comparison component 220 compares a valueassociated with a characteristic the defect 402 to known defects in thedefects database 216 to identify that the defect 402 corresponds tosegregation of the aggregate. The command component 222 generates one ormore commands 404 that, when executed by the paving machine 100, atleast partially remedies the defect 402. In this case, the commands 404include adjusting the auger height, adjusting the auger speed, adjustinga position of the feed sensor, adjusting the mounting distance of thefeed sensor, and so forth. Two or more of the commands 404 are providedby the mat defect ID component 214 to the paving machine 100 and/or theinteractive mat defect component 212 together if more than one commandshould be executed in association with the paving machine 100 to remedythe defect 402. Alternatively or additionally, two or more of thecommands 404 are provided by the mat defect ID component 214 to thepaving machine 100 and/or the interactive mat defect component 212 inseries, responsive to a determination that a first command did notremedy the defect. FIGS. 5 and 6 illustrate examples of components ofthe paving machine 100 that may be altered to remedy a defect inaccordance with the commands 404.

FIG. 5 is a schematic illustration 500 of the paving machine 100 of FIG.1 and associated components of the paving machine 100 depositing pavingmaterial on a paving surface 122. Generally, different mixes of pavingmaterial 110 react differently to adjustment of a height of the augerassembly 114. Therefore, the auger height may need to be adjusted up ordown based on mix type of the paving material 110 and an appearance ofthe mat 120 after paving has begun.

The auger assembly 114 has a radius 504 extending from a perimeter ofthe auger assembly 114 to a center of an auger shaft 506 of the augerassembly 114. In some examples, the radius 504 of the auger assembly 114is equal to approximately 8 inches. A height 508 corresponds to a height(e.g., depth) of the mat 120 before the mat 120 is compacted, and theheight 508 extends from an uppermost surface of the paving surface 122to an uppermost surface of the mat 120. In some examples, the height 508is equal to between approximately 2 inches and approximately 4 inches. Aheight 510 corresponds to a distance between the uppermost surface ofthe mat 120 and the perimeter of the auger assembly 114 before the mat120 is compacted. In some examples, the height 510 is equal to betweenapproximately 2 inches and approximately 4 inches. In some examples, aheight 512 of the auger assembly 114 is set to equal the sum of theradius 504, the height 508, and the height 510. Using the exampledimensions just described, the height 512 would be set to betweenapproximately 12 inches and approximately 16 inches. Other auger radii,auger heights, and/or other dimensions are also considered, and may varybased on different auger sizes, different paving materials, and soforth.

In some cases, the height 512 of the auger assembly 114 may need to beadjusted after paving starts. For example, if the height 512 of theauger assembly 114 is set lower than a prescribed ratio (e.g., 2:1)based on material type and characteristics of the auger assembly 114,segregation of the mat 120 as illustrated by the defect 402 can result.The defect 402 may be more severe when the paving material 110 compriseslarger stone mix. In some examples, the command 404 instructs the pavingmachine 100 (and/or an operator of the paving machine 100 via theinteractive mat defect component 212 of the mobile device 208) to travelat least a distance equivalent to a full length of the paving machine100 before evaluating whether the defect 402 has resolved.

FIG. 6 is a schematic illustration 600 of a feed sensor 602 which isincluded on the paving machine 100 to determines a distance to thepaving material 110. The feed sensor 602 may be a mechanicalpaddle-type, a sonic sensor, or other type of sensor. In examples, thefeed sensor 602 generates a signal that indicates a distance of the feedsensor 602 from the paving material 110 at an outboard end of an auger604, where the paving material 110 generally moves in a steady pattern.The feed sensor 602 should be positioned such that the feed sensor 602is sensing the active pile of the paving material 110, for instance,about 18 inches (46 cm) away from a last segment of the auger 604. Insome examples, if the feed sensor 602 is a paddle-type sensor, a paddlearm of the feed sensor 602 should be positioned at substantially a45-degree angle from the paving material 110 at the 18-inch distance. Ifthe feed sensor 602 is positioned too close (e.g., less than a thresholdamount based on a composition of the paving material 110) to the auger604, the signal generated by the feed sensor 602 will be affected by a“wave” action of the paving material 110 coming off the auger 604 andoperation of the paving machine 100 will be erratic. If the feed sensor602 is too far away (e.g., greater than a threshold amount based on acomposition of the paving material 110) from the auger 604, the signalgenerated by the feed sensor 602 may cause the feeder system to overfillthe auger chamber. Either erratic operation of the paving machine 100 oroverfilling of the auger chamber can cause segregation as illustrated bythe defect 402.

In an example in which the feed sensor 602 is a sonic sensor, the feedsensor 602 generates a pulse 606 of sound that travels to the pavingmaterial 110 and is reflected back to the feed sensor 602 when the feedsensor 602 is aimed perpendicular to a face of the paving material 110.The feed sensor 602 measures the time it takes for the pulse 606 totravel to the paving material 110 and back, and the controller 154calculates the distance from the feed sensor 602 to the paving material110 based on the measured time. In the example in which the distancebetween the feed sensor 602 and the paving material 110 that willprevent segregation is 18 inches (46 cm), the operational range of thefeed sensor 602, when the feed sensor 602 is a sonic sensor, may rangefrom less than 12 inches (30 cm) resulting in the controller 154 causingthe feed of paving material 110 to fully shut off, to 32 inches (81 cm)resulting in the controller 154 causing the feed of paving material 110to be fully on. The controller 154 modulates the flow of the pavingmaterial 110 according to the sensed distance between 12 inches and 32inches. Accordingly, a default distance of 18 inches from the feedsensor 602 to the paving material 110 will result in an easy-to-controlmix height at the end of the auger 604.

However, in some instances, the feed sensor 602 may become misalignedfrom the paving material, which causes a pulse 608 to bounce away fromthe feed sensor 602 rather than directly back to the feed sensor 602.When the feed sensor 602 does not sense the pulse 608 returning, thefeed sensor 602 may function erratically or not at all. Consequently,the amount of the paving material 110 being delivered may have incorrectconsistency, and/or may cause the mat 120 to be too thick or too thin,thus resulting in segregation represented by the defect 402.Accordingly, the commands 404 include adjusting the feed sensor positionand/or adjusting the feed sensor mounting distance to resolve the issuesjust described that may arise associated with the feed sensor 602 andcausing the segregation represented by the defect 402.

FIG. 7 is another schematic illustration of an image 700 indicative ofthe mat 120 having a defect 702, in this case short waves/ripples in themat 120 across a full width of the mat 120. Ripples in the mat 120 asindicated in the image 700 may result from a variety of causes, such asa screed riding at a high angle of attack, the screed riding on liftcylinders, screed plates being loose, and/or grade control improperlyset, among others.

The mat defect ID component 214 receives the image 700 (and/or othersensor data corresponding to the mat 120, as described above), and usesthe feature detector 218 to detect, for instance, ridges and/or edgescorresponding to locations of the defect 702 in the image 700. Thecomparison component 220 compares a value associated with acharacteristic of the defect 702 to known defects in the defectsdatabase 216 to identify that the defect 702 corresponds to ripples inthe mat 120. The command component 222 generates one or more commands704 that, when executed by the paving machine 100, at least partiallyremedies the defect 702. In this case, the commands 704 includeadjusting the components of the mix of the paving material 110,adjusting the grade, checking parts of the screed portion 118, and soforth. Two or more of the commands 704 are provided by the mat defect IDcomponent 214 to the paving machine 100 and/or the interactive matdefect component 212 together if more than one command should beexecuted in association with the paving machine 100 to remedy the defect702. Alternatively or additionally, two or more of the commands 704 areprovided by the mat defect ID component 214 to the paving machine 100and/or the interactive mat defect component 212 in series, responsive toa determination that a first command did not remedy the defect.

FIG. 8 is yet another schematic illustration of an image 800 indicatingthe mat 120 having a defect 802. In this example, the defect 802corresponds to a surface texture of the mat 120 being non-uniform. Insome cases, non-uniform surface texture of the mat 120 as indicated inthe image 800, where the non-uniformity occurs behind the extender(s)804 of the paving machine 100, may result from the angle of attack ofthe extenders being too low.

The mat defect ID component 214 receives the image 800 (and/or othersensor data corresponding to the mat 120, as described above), and usesthe feature detector 218 to detect, for instance, an edge correspondingto a location of the defect 802 in the image 800. The comparisoncomponent 220 compares a value associated with a characteristic of thedefect 802 to known defects in the defects database 216 to identify thatthe defect 802 corresponds to non-uniform mat texture occurring behindthe extenders. The command component 222 generates one or more commands806 that, when executed by the paving machine 100, at least partiallyremedies the defect 802. In this case, the command 806 includesadjusting an angle of the extender 804, although may include othercommands as well as described herein.

FIG. 9 is still another schematic illustration of an image 900indicating the mat 120 on having a defect 902. In this example, thedefect 902 corresponds to a “torn spot” on the surface of the mat 120.For instance, the paving machine 100 is set up with grade control onboth side of the screed portion 118 using an averaging ski on each sideof the screed portion 118. The averaging skis control the screed portion118 such that the screed portion 118 fills in low spots and scalps offhigh spots of the paving material 110 deposited on the paving surface122. In some cases, the paving machine 100 reaches a location at which acold planer of the paving machine 100 left a high spot in the grade,resulting in the screed portion 118 laying a thinner mat 120 thanoptimal. The ratio of thickness of the mat 120 to the size of theaggregate in the paving material 110 goes below a threshold ratio, suchas 2:1, resulting in the mat 120 having a torn appearance. Ideally, thehigh spot in the grade should be corrected before paving.

The mat defect ID component 214 receives the image 900 (and/or othersensor data corresponding to the mat 120, as described above), and usesthe feature detector 218 to detect, for instance, an edge and/or a blobcorresponding to locations of the defect 902 in the image 900. Thecomparison component 220 compares a value associated with acharacteristic of the defect 902 to known defects in the defectsdatabase 216 to identify that the defect 902 corresponds to a torn mat120. The command component 222 generates one or more commands 904 that,when executed by the paving machine 100, at least partially remedies thedefect 902. In this case, the command 904 includes locating theirregularity in the grade that caused the high spot and resulting tornmat 120, although may include other commands as well as describedherein.

FIG. 10 is an additional schematic illustration of an image 1000indicating the mat 120 having a defect 1002. In this example, the defect1002 corresponds to the paving material 110 being spilled on the mat120. In some cases, spilling the paving material 110 on the mat 120 asindicated in the image 1000 can cause high spots that result in tearingof the mat 120 after the spreader of the paving machine 100 travels overthe spills. The spilled paving material 110 should be cleaned before thespreader passes over the spilled paving material 110 to prevent highspots.

The mat defect ID component 214 receives the image 1000 (and/or othersensor data corresponding to the mat 120, as described above), and usesthe feature detector 218 to detect, for instance, a blob correspondingto locations of the defect 1002 in the image 1000. The comparisoncomponent 220 compares a value associated with a characteristic of thedefect 1002 to known defects in the defects database 216 to identifythat the defect 1002 corresponds to paving material 110 being spilled onthe mat 120. The command component 222 generates one or more commands1004 that, when executed by the paving machine 100, at least partiallyremedies the defect 1002. In this case, the command 1004 includescleaning up the spilled paving material 110, although may include othercommands as well as described herein. In this case, the command 1004 maybe escalated using sound, haptics, sending a notification to multiplecrew members, or the like in order to promote the time-sensitive natureof cleaning the spilled paving material 110 before the spreader reachesthe location of the spill.

FIG. 11 is a further schematic illustration of an image 1100 indicatingthe mat 120 having a defect 1102. In this example, the defect 1102corresponds to the mat 120 being torn inside of the path of the screedportion 118 of the paving machine 100. In some cases, the defect 1102 asrepresented in the image 1100 may result from the screed extensionheight being too low. Alternatively or additionally, the screedextension being too high may result in the line corresponding to thedefect 1102 being in alignment with an outer edge of the main screed.

The mat defect ID component 214 receives the image 1100 (and/or othersensor data corresponding to the mat 120, as described above), and usesthe feature detector 218 to detect, for instance, an edge correspondingto a location of the defect 1102 in the image 1100. The comparisoncomponent 220 compares a value associated with a characteristic of thedefect 1102 to known defects in the defects database 216 to identifythat the defect 1102 corresponds to the mat 120 being torn in the pathof the screed portion 118. The command component 222 generates one ormore commands 1104 that, when executed by the paving machine 100, atleast partially remedies the defect 1102. In this case, the commands1104 include adjusting the screed extension height, adjusting a slope ofthe screed extension, and so forth. In some cases, two or more of thecommands 1104 are provided by the mat defect ID component 214 to thepaving machine 100 and/or the interactive mat defect component 212 inseries, responsive to a determination that a first command did notremedy the defect. In an illustrative example, the mat defect IDcomponent 214 provides a first command to the interactive mat defectcomponent 212 instructing an operator to lower the screed extensionuntil the defect 1102 disappears from the mat 120. If the defect 1102reappears, the mat defect ID component 214 determines that the slope ofthe screed extension is incorrect. Accordingly, the mat defect IDcomponent 214 provides a second command to the interactive mat defectcomponent 212 to use a slope extension switch to remove the defect 1102.Additionally, in some cases, the mat defect ID component 214 alsoprovides a third command with the second command to have the operatorcheck the position of the extension slope stop to prevent continuousproblems with slope height.

FIG. 12 is still another schematic illustration of an image 1200indicating the mat 120 having a defect 1202. In this example, the defect1202 corresponds to a bump caused when a haul truck bumps into thepaving machine 100, causing the screed portion 118 to gouge the mat 120.

The mat defect ID component 214 receives the image 1200 (and/or othersensor data corresponding to the mat 120, as described above), and usesthe feature detector 218 to detect, for instance, an edge and/or a ridgecorresponding to locations of the defect 1202 in the image 1200. Thecomparison component 220 compares a value associated with acharacteristic of the defect 1202 to known defects in the defectsdatabase 216 to identify that the defect 1202 corresponds to a gougemade by the screed portion 118 when a haul truck bumps into the pavingmachine 100 when depositing the paving material 110. The commandcomponent 222 generates one or more commands 1204 that, when executed bythe paving machine 100, at least partially remedies the defect 1202. Inthis case, the commands 1204 include instructing an operator of the haultruck to stop short of the paving machine 100, instructing an operatorof the paving machine 100 to move the paving machine 100 to the haultruck to unload the paving material 110, and so forth. In this case, thecommands 1204 are sent to both an electronic device associated with theoperator of the haul truck and an electronic device associated with theoperator of the paving machine 100 substantially simultaneously toensure that both operators know what to do to remedy the defect 1202.

FIG. 13 is a flow chart depicting a method 1300 of generating a commandto remedy an identified paving mat defect in accordance with an exampleof the present disclosure. The example method 1300 is illustrated as acollection of steps in a logical flow diagram, which representsoperations that can be implemented in hardware, software, or acombination thereof. In the context of software, the steps representcomputer-executable instructions stored in memory. When suchinstructions are executed by, for example, the controller 154, suchinstructions cause the controller 154, various components of the controlsystem 200, and/or the paving machine 100, generally, to perform therecited operations. Such computer-executable instructions compriseroutines, programs, objects, components, data structures, and the likethat perform particular functions or implement particular abstract datatypes. The order in which the operations are described is not intendedto be construed as a limitation, and any number of the described stepscan be combined in any order and/or in parallel to implement theprocess. For discussion purposes, and unless otherwise specified, themethod 1300 is described with reference to the paving machine 100 ofFIG. 1 and the control system 200 of FIG. 2.

At 1302, the mat defect ID component 214 receives sensor data, such asimage data, indicative of a paved surface, such as from the pavingmachine 100 and/or the mobile device 208 via the network 206. While themethod 1300 generally makes reference to image data, the controller 154may alternatively or additionally receive other data types as well, suchas video, LIDAR data, RADAR data, infrared data, and the like from thecamera 134 and/or the camera 210, the LIDAR sensor 150, the RADAR sensor152, the temperature sensors 136, 138, 140, and so on.

At 1304, the mat defect ID component 214 identifies a defect associatedwith the paved surface. For example, the feature detector 218 computesabstractions of the image data and makes local decisions at image pointsin the image data regarding whether the image point includes an imagefeature or not. The feature detector 218 detects features such as edges,ridges, blobs, or other feature types. Additionally, the mat defect IDcomponent 214 uses the features identified by the feature detector 218to locate, within an image, a location of a potential mat defect of themat 120.

At 1306, the mat defect ID component 214 determines that the defect isof a defect type. In some examples, the comparison component 220compares a potential mat defect in the image data to known defects inthe defect database 216. In examples, the comparison component 220determines a value associated with the detected feature in the imagedata and based in part on the feature detected by the feature detector218. The comparison component 220 compares the value to known values(and/or value ranges) associated the known defects in the defectdatabase 216. The comparison component 220 relies on one or more of avariety of characteristics of defects and their associated values todetermine the presence of a mat defect from sensor data, such as sizevalues (e.g., length, width, height, depth, and the like of thesuspected defect), temperature values, distance values and/or anglevalues (e.g., relative to the paving machine 100 or a particular part ofthe paving machine 100, relative to an extent of the mat 120, and thelike), and so on. In some examples, the defect database 216 includesvalue ranges associated with characteristics that indicate the presence(and/or absence) of a defect in the mat 120. For example, the defectdatabase 216 includes a threshold size of 2 square feet for largeaggregate consolidation in the mat 120. The comparison component 220compares a suspected aggregate segregation defect in image data to thethreshold size stored in the defect database 216 to determine whetherthe mat 120 includes the aggregate segregation defect.

At 1308, the mat defect ID component 214 generates a command associatedwith the defect that, when executed by a machine such as the pavingmachine 100, at least partially remedies the defect. For instance, thecommand component 222 maps the defect type to one or more commandsstored in the defects database 216 known to at least partially remedydefects of the defect type. In some cases, the command component 222determines multiple commands to be executed by one or more machines,either together (e.g., substantially simultaneously) or in parallel,that at least partially remedy the defect. The command component 222determines which machines to provide the command(s) to, such as thepaving machine 100, the mobile device 208, a haul truck, or anothermachine based on how the command is to be executed and machines mappedto commands in the defect database 216. For example, the command 1004 isprovided to the mobile device 208 to instruct a crew member to cleanspilled paving material 110, while the command 806 is provided directlyto the paving machine 100 to adjust the extender angle automatically andwithout user intervention.

At 1310, the mat defect ID component 214 determines whether the defectis remedied. For example, the mat defect ID component 214 comparessensor data received subsequent to providing the command to the pavingmachine 100 and/or the mobile device 208 to sensor data in the defectdatabase 216 corresponding to defect-free paving mats, and paving matshaving the suspected defect, as described above. If the mat defect IDcomponent 214 determines that the defect is remedied (e.g., “Yes” at1310), at 1312 the mat defect ID component 214 continues to receiveimage data (or other sensor data) from the paving machine 100 and/or themobile device 208 and continues to monitor for defects. If the matdefect ID component 214 determines that the defect is not remedied(e.g., “No” at 1310), the method 1300 returns to operation 1308. Forexample, the mat defect ID component 214 generates a different commandthan the original command to remedy the defect, which includes differentinstructions to at least partially remedy the defect. The mat defect IDcomponent 214 provides the different command to an electronic devicesuch as the paving machine 100 and/or the mobile device 208, similar tothe discussion above.

INDUSTRIAL APPLICABILITY

The present disclosure provides systems and methods for correcting adefect associated with a mat 120 of paving material 110 formed at aworksite. Such systems and methods may be used to achieve better pavingand compacting performance by remedying defects earlier and with greateraccuracy. Additionally, such systems and methods may be used to improveefficiencies in remedying defects by not having to rely upon pavingoperators who are inexperienced with the variety of defects that mayoccur and how to resolve such defects. As noted above with respect toFIGS. 1-13, an example method 1300 of correcting paving mat defects mayinclude receiving, with a controller 154, image data indicative of apaved surface 122 and identifying, with the controller 154 and based atleast in part on the image data, a defect associated with the pavedsurface 122, the defect having a characteristic that is characterized bya value. Such an example method 1300 also includes determining, with thecontroller 154, that the value is within a value range corresponding toa defect type, and determining, based at least in part on the valuebeing within the value range, that the defect is of the defect type.Such an example method 1300 further includes generating a command 404associated with the defect and based at least in part on determiningthat the defect is of the defect type. In the example method 1300, thecommand 404 identifies a proposed machine activity which, when executedby a machine 100 on the paved surface 122, at least partially remediesthe defect. Such an example method 1300 further includes providing thecommand 404 to an electronic device 208 via a network 206.

By comparing values associated with suspected paving mat defectsdetermined from sensor data, such as image data, with known values fordifferent defect types, defects can be remedied more accurately andefficiently than previous techniques. For example, the mat defect IDcomponent 214 can provide a command 404 to an electronic device 208associated with an operator of the paving machine 100, which instructsthe operator on how to change a setting of the paving machine 100 toremedy the detected defect. Alternatively or additionally, the matdefect ID component 214 can provide a command 404 directly to the pavingmachine 100 to execute the command 404 to remedy the defect without userintervention, allowing the paving machine 100 to autonomously correctdefects while a paving project is underway. Such information may also beused by the paving material production plant to more closely managepaving material compositions and/or to optimize the scheduling of haultruck deliveries based on a type of defect detected and an estimatedtime to remedy the defect. Thus, the example systems and methodsdescribed above may provide considerable cost savings, and may reducethe time and labor required for various paving activities at theworksite.

While aspects of the present disclosure have been particularly shown anddescribed with reference to the embodiments above, it will be understoodby those skilled in the art that various additional embodiments may becontemplated by the modification of the disclosed machines, systems andmethods without departing from the spirit and scope of what isdisclosed. Such embodiments should be understood to fall within thescope of the present disclosure as determined based upon the claims andany equivalents thereof.

1.-20. (canceled)
 21. A method, comprising: receiving image dataindicative of a paved surface; identifying, based at least in part onthe image data, a defect associated with the paved surface, the defecthaving a characteristic that is characterized by a value; determiningthat the value is within a range of values corresponding to a defecttype; determining, based at least in part on the value being within therange of values, that the defect corresponds to the defect type;determining, based at least in part on the defect corresponding to thedefect type, a command associated with a machine at least partiallyremedying the defect; and sending the command to at least one of anelectronic device or the machine, wherein the command, when executed,causes the machine to at least partly remedy the defect.
 22. The methodaccording to claim 21, wherein the image data is received from animaging device associated with the at least one of the electronic deviceor the machine.
 23. The method according to claim 21, wherein the imagedata comprises first image data, the method further comprising:receiving second image data indicative of the paved surface; andidentifying, based at least in part on the second image data, the defectassociated with the paved surface, wherein determining that the defectcorresponds to the defect type is further based at least in part onidentifying the defect in the second image data.
 24. The methodaccording to claim 23, wherein the first image data is generated by afirst imaging device, and the second image data is generated by a secondimaging device substantially simultaneously with the first image data.25. The method according to claim 21, wherein the defect type comprisesa first defect type and the range of values comprises a first range ofvalues, the method further comprising: determining that the value iswithin a second ranges of values corresponding to a second defect type;generating a notification that includes: a first image associated withthe first defect type, a second image associated with the second defecttype, and a request for a user to select whether the defect correspondsto the first defect type or the second defect type; sending thenotification to the electronic device; and receiving an indication fromthe electronic device, the indication being indicative of the defectcorresponding to the first defect type, wherein the command isdetermined based at least in part on the indication.
 26. The methodaccording to claim 21, wherein the image data comprises first image dataand the defect comprises a first defect, the method further comprising:determining, based at least in part on second image data, a thresholdsize of a second defect known to be of the defect type indicated in thesecond image data, wherein determining that the first defect is of thedefect type is further based at least in part on a size of the firstdefect satisfying the threshold size.
 27. A machine, comprising: asensor; one or more processors; and one or more non-transitorycomputer-readable media storing instructions that, when executed by theone or more processors, cause the one or more processors to performoperations comprising: receiving, from the sensor, sensor dataindicative of a paved surface at which the machine operates;identifying, based at least in part on the sensor data, a defectassociated with the paved surface; determining a value characterizingthe defect; determining that the value is within a range of valuescorresponding to a type of defect present on the paved surface;determining, based at least in part on the value being within the rangeof values, that the defect corresponds to the type of defect;determining a setting of the machine associated with at least partiallyremedying the defect; and operating the machine in accordance with thesetting, wherein operating the machine in accordance with the settingcauses the defect to be at least partly remedied.
 28. The machineaccording to claim 27, wherein the sensor data comprises first sensordata and the setting comprises a first setting, the operations furthercomprising: receiving, from the sensor, second sensor data indicative ofthe paved surface; and identifying, based at least in part on the secondsensor data, an absence of the defect associated with the paved surface.29. The machine according to claim 27, wherein the sensor comprises afirst sensor, the sensor data comprises first sensor data, the defectcomprises a first defect, and the machine further comprises a secondsensor, the operations further comprising: receiving, from the secondsensor, second sensor data indicative of the paved surface; identifying,based at least in part on the second sensor data, a second defectassociated with the paved surface; determining that the first defect andthe second defect a same type of defect, wherein determining the defectis further based at least in part on determining that the first defectand the second defect are the same type of defect.
 30. The machineaccording to claim 27, wherein the sensor data comprises image data, andidentifying the defect associated with the paved surface comprises:inputting at least a portion of the image data into a machine-learnedmodel trained to identify defects associated with paved surfaces; andreceiving, from the machine-learned model, a location of the defectwithin the image data, wherein the defect is identified based at leastin part on the location of the defect in the image data.
 31. The machineaccording to claim 27, wherein determining the setting comprises:inputting the type of defect into a machine-learned model trained todetermine remedies for defects within paved surfaces; and receiving,from the machine-learned model, a remedy for the defect, wherein thesetting is based at least in part on the remedy.
 32. The machineaccording to claim 27, the operations further comprising: causing outputof a notification associated with the defect; and receiving a responseindicating that the defect corresponds to the type of defect, whereindetermining that the defect corresponds to the type of defect is furtherbased at least in part on the response.
 33. A method, comprising:receiving sensor data indicative of a paved surface at a worksite;identifying, based at least in part on the sensor data, a featureassociated with the paved surface; identifying, based at least in parton the feature, a defect within the paved surface; determining a type ofdefect associated with the defect; determining, based at least in parton the type of defect, an action associated with a machine at leastpartially remedying the defect; causing output of an indicationassociated with at least one of the defect, the type of defect, or theaction; and causing the action to be performed at least in part by themachine.
 34. The method according to claim 33, wherein the indicationcomprises a first indication, the method further comprising: causing anoutput of a second indication associated with the defect; and receivinga third indication associated with a confirmation of the defect, whereinthe type of defect is determined based at least in part on the thirdindication.
 35. The method according to claim 33, wherein the sensordata comprises first sensor data and the defect comprises a firstdefect, the method further comprising: receiving second sensor dataindicative of the paved surface at the worksite; identifying, based atleast in part on the second sensor data, a second defect within thepaved surface; determining that the first defect and the second defectrepresent a similar type of defect within the paved surface, and whereindetermining the action is further based at least in part on the firstdefect and the second defect representing the similar type of defect.36. The method according to claim 33, further comprising: inputting thesensor data into a machine-learned model; receiving, as output from themachine-learned model, a score representing a likelihood that the defectis present within the paved surface; and determining that the scoresatisfies a threshold score, wherein identifying the defect within thepaved surface is further based at least in part on the score satisfyingthe threshold score.
 37. The method according to claim 33, wherein theindication comprises a first indication, the method further comprising:sending at least a portion of the sensor data to a computing resource;receiving a second indication from the computing resource, the secondindication indicating at least one of: the defect, the type of defect,or the action, wherein causing the action to be performed is based atleast in part on the second indication.
 38. The method according toclaim 33, wherein the sensor data comprises first sensor data, themethod further comprising: identifying second sensor data associatedwith the type of defect; determining, based at least in part on thesecond sensor data, a range of values associated with the type ofdefect; determining a value associated with the feature; and determiningthat the value is within the range of values, wherein determining thetype of defect associated with the defect is based at least in part onthe value being within the range of values.
 39. The method according toclaim 38, wherein the value corresponds to a size of the defect, atemperature of the paved surface, or a location of the defect on thepaving surface.
 40. The method according to claim 33, wherein: theaction is associated with a setting of a machine operating at a worksiteassociated with the paving surface; and causing the action to beperformed comprises causing the machine to implement the setting, thesetting at least partially remedying the defect.